Model-based security testing: a taxonomy and systematic classification

On this level, an approach is evaluated on the basis of non-executable test cases. For instance, if generated test cases are sequences of non-executable nodes of a transition-based model, then evidence can already be measured (efficiency can, for example, be measured on the basis of a cost or time model). But, the concrete effect of the approach on the deployed system under test cannot be measured. Executable: On this level, an approach is evaluated on the basis of test cases, which are executable against the system under test. Abstract test cases are often generated as an intermediate step. Compared with the abstract level, the set of possible evidence measures is not limited. 4. SYSTEMATIC SELECTION AND CLASSIFICATION OF PUBLICATIONS ON MODEL-BASED SECURITY TESTING This section describes the systematic selection of model-based security testing publications and their classification according to the filter and evidence criteria defined in the previous section. Threats to validity of the selection and classification procedure are discussed. The resulting classification of MBST approaches covered in the selected publications is then used in the next section to indicate the adequacy of the defined criteria and for further discussions. Copyright © 2015 John Wiley & Sons, Ltd. Softw. Test. Verif. Reliab. (2015) DOI: 10.1002/stvr MODEL-BASED SECURITY TESTING 4.1. Selection of publications The selection of relevant peer-reviewed primary publications comprises the definition of a search strategy and paper selection criteria as well as the selection procedure. 4.1.1. Search strategy. The conducted search of the relevant publications included an automatic search in the following digital libraries: IEEE Digital Library (http://ieeexplore.ieee.org/); ScienceDirect (http://www.sciencedirect.com/); Springer Link (http://link.springer.com); ACM Digital Library (http://portal.acm.org/); and Wiley (http://onlinelibrary.wiley.com/). These libraries were chosen because they cover the most relevant sources in software and security engineering [35]. The search string applied to the digital libraries is as follows: ( "model based" OR automata OR "state machine" OR "specification based" OR policy OR policies OR "threat model" OR mutation OR risk OR fuzzing ) AND ( security OR vulnerability OR privacy OR cryptographic ) AND ( test OR testing ) This search string combines terms for (security) models and artefacts with terms for security and testing. It is based on the search string of Dias-Neto and Travassos used in their systematic review on model-based testing [6]. The search string was piloted and iteratively improved to cover all relevant publications from a reference database on model-based security testing from a research project that was additionally extended by key publications known by the three researchers who performed the search and are all experts in the field of model-based security testing. The reference database contained 76 relevant publications from all five searched digital libraries. The final version of the search string covered all relevant publications of this reference database and therefore had a recall, that is, ratio of the retrieved relevant items and all existing relevant items [36], of 100 percent. The start year of the search was set to 1996, which is the publication year of the earliest modelbased security testing approach listed in the survey of Dias-Neto and Travassos [6]. The last year of inclusion is publications within 2013. The search fields title, abstract and keywords were considered. 4.1.2. Paper selection. On the set of papers found with the search strategy, suitable inclusion and exclusion criteria for selecting relevant primary publications were applied. Included were peerreviewed publications written in English that cover a model-based security testing approach as defined in Section 1. Excluded were related approaches to classical penetration testing, static analysis, monitoring and testing of properties such as robustness, safety or trust; these do not cover a model-based security testing approach according to the definition in this article. As illustrated in Figure 3, the initial search on the basis of the search strategy delivered the following results. In the ACM Digital Library, 3038 publications were found, in the IEEE Digital Library 1769, in Science Direct 111, in Springer Link 1000 and in Wiley 10. On the basis of this result set, the relevant publications were selected in three phases as shown in Figure 3 by three researchers. In phase 1, duplicates and irrelevant papers according to the selection criteria were removed on the basis of the title. In this phase, 5268 papers were excluded from the originally retrieved publications leaving 660 filtered papers. In phase 2, irrelevant papers according to the selection criteria were removed on the basis of the abstract. In this phase, 336 papers were excluded from the filtered publications leaving 324 papers for full text reading. In phase 3, irrelevant papers according to the selection criteria were removed on the basis of the full text. Papers with a length below four pages were additionally removed. In this phase, 205 papers Copyright © 2015 John Wiley & Sons, Ltd. Softw. Test. Verif. Reliab. (2015) DOI: 10.1002/stvr